Measuring data reliability for preventive services in electronic medical records

BACKGROUND Improvements in the quality of health care services are often measured using data present in medical records. Electronic Medical Records (EMRs) contain potentially valuable new sources of health data. However, data quality in EMRs may not be optimal and should be assessed. Data reliability (are the same data elements being measured over time?) is a prerequisite for data validity (are the data accurate?). Our objective was to measure the reliability of data for preventive services in primary care EMRs during the transition to EMR. METHODS Our data sources were randomly selected eligible patients' medical records and data obtained from provincial administrative datasets. Eighteen community-based family physicians in Toronto, Ontario that implemented EMRs starting in 2006 participated in this study. We measured the proportion of patients eligible for a service (Pap smear, screening mammogram or influenza vaccination) that received the service. We compared the change in rates of selected preventive services calculated from the medical record audits with the change in administrative datasets. RESULTS In the first year of EMR use (2006) services decreased by 8.7% more (95% CI -11.0%- - 6.4%, p < 0.0001) when measured through medical record audits as compared with administrative datasets. Services increased by 2.4% more (95% CI 0%-4.9%, p = 0.05) in the medical record audits during the second year of EMR use (2007). CONCLUSION There were differences between the change measured through medical record audits and administrative datasets. Problems could include difficulties with organizing new data entry processes as well as continued use of both paper and EMRs. Data extracted from EMRs had limited reliability during the initial phase of EMR implementation. Unreliable data interferes with the ability to measure and improve health care quality.